Repository navigation
test(hermitcrab): opt-in stress grammars for optimization parity (not conformance) - #578
johnml1135 wants to merge 3 commits into
Conversation
Adds the round-doc section "Combined stack and R2's cost": the full stack against master (work, unification, matcher per grammar), additivity (near-additive on Mbugwe, overlapping on Sena, super-additive on Aweti), R2's root cause (its own probe, low prune rate) and the witness-reuse fix folded into #572 @ 01bdf5a, the 1-4 point residual, and the landing decision: #572 is a draft and lands only on top of #577 and #574. Stress grammars are PR #578 (opt-in, [Explicit], not run in CI). Ledger: row 40 (R2) Outcome records the fold and the landing decision; new row 47 records the combined stack measurement. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
| RunReport report = Runner.RunSelfCheck( | ||
| fixtures, | ||
| includePathological: true, | ||
| engineCapabilities: new HashSet<string>(StringComparer.Ordinal) { "phonology", "morpher-settings" }, |
There was a problem hiding this comment.
FYI: F1. This copies the capability set instead of using the engine's own, so once SelfCheckEngine gains a capability that a stress grammar requires, this test skips that fixture, but the README's CLI command still runs it. new SelfCheckEngine().Capabilities stays in step (SelfCheckEngine.cs:22).
| proposeOutput: TextWriter.Null | ||
| ); | ||
| Assert.That(report.Failed, Is.EqualTo(0), string.Join("; ", report.Results.Select(r => r.Reason))); | ||
| Assert.That(report.Passed, Is.EqualTo(1)); |
There was a problem hiding this comment.
FYI: F2. A skipped fixture fails here with only Expected: 1 But was: 0. The skip reason (Runner.cs:98-100, "missing [...]") is in report.Results but is not passed to this assert. Pass the same Reason join used at line 39 as the message.
| [Category("Stress")] | ||
| public sealed class StressFixtureParityTests | ||
| { | ||
| private static string StressRoot() |
There was a problem hiding this comment.
FYI: F3. This is a seventh copy of the ancestor walk that six test files here already have as RepositoryRoot(), which anchors on conformance/constructs.txt. This copy stops at the first ancestor with any conformance-stress directory. Path.Combine(RepositoryRoot(), "conformance-stress") would anchor on the same root as the others.
| @@ -0,0 +1,554 @@ | |||
| # kind: stress (timing and optimization parity fixture, not conformance; see conformance-stress/README.md) | |||
| # StressAthabaskanPrefixComplex: dense-exclusion stress fixture, generated by gen_dense_exclusions.py. The expected parses | |||
There was a problem hiding this comment.
FYI: F4. The provenance can't be reproduced: gen_dense_exclusions.py, gen_excluded_cooccurrence.py, Morpher.PruneExcludedCoOccurrence, PruneByTemplateReplay and "the E1 cutoff" are not in the tree (searched *.cs, *.py, *.md outside conformance-stress/). The same applies to bantu-extension-chain, excluded-cooccurrence, required-template-slot and prefix-class-density. Either link the branch or PR that has them, or say the expectations are master's output, as the README does.
This comment has been minimized.
This comment has been minimized.
| fieldworks_producible_notes: "research stress fixture; not authored through FieldWorks" | ||
| budget_ms: 60000 | ||
| words: | ||
| - word: wnonemileila |
There was a problem hiding this comment.
Minor: F5. No expected parse uses a fusion allomorph, so the parity check cannot catch an impure-slot look-ahead that wrongly prunes one. All 16 words shaped like a fused subrule output fail on master, for example tkokaawabani = fsub_subj_*_to on kaawabani. That accounts for 28 no-parse words here against 12 in bantu-tam-templates. The fsub_* output (grammar.xml:46) also is not vowel fusion: it copies the first stem segment on both sides of the inserted vowel. Could you make the fused forms parse on master, or note in the README that they never do?
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
…nce-stress, outside the conformance suite and CI Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…for the impure-slot look-ahead stacked on #576. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he worst case for the supply-set look-ahead stacked on #579. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
262ab30 to
cf84539
Compare
df134b0 to
a37fa2a
Compare
| note: >- | ||
| Parse by allomorph selection: each prefix allomorph is chosen by the segment that follows it, the suffix by the segment before it, and an earlier conditioned allomorph whose context holds blocks the fallback. | ||
| parses: | ||
| - signature: "R13+TAM|apani" |
There was a problem hiding this comment.
Major: F6. Since the rebase, the parity test fails on this PR's own head, so the expectations no longer match master. dotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --filter TestCategory=Stress gives 3 failed and 15 passed, the same on a rerun. Here 10/40 words mismatch: apani expects [R13+TAM] but gets [R13+CONC+TAM | R13+TAM]. env-austronesian-nasal-prefix (1/40, mempendasi) and env-zero-allomorphs (2/40, kanu, kanupi) also fail. All three are environment-conditioned allomorph grammars. That fits #548 (filter synthesis by allomorphs, merged to master on 10-27), but I did not bisect it (unverified). Please regenerate these three from current master.
Public API, target framework, package, machine.py parity: None verified. Findings: F1 open, F2 open, F3 open, F4 open, F5 open, F6 new (cause unverified). Reviewed at a37fa2a |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## integrate-conformance-framework #578 +/- ##
================================================================
Coverage 75.09% 75.10%
================================================================
Files 556 556
Lines 56531 56531
Branches 7719 7719
================================================================
+ Hits 42454 42459 +5
+ Misses 12215 12211 -4
+ Partials 1862 1861 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Start here:
conformance-stress/README.md-- it lists the 18 grammars, what each exercises and which optimization it was built for.Status: stacked on #480 — merge in order #556 → #500 → #480 → #578.
Status: draft. Stacked on #480 (open); merge #480 first, and the base
integrate-conformance-frameworkhas moved since this was cut. CI is red: the Build jobs (ubuntu and windows) failEveryConformanceFixturePassesSelfCheck, and the Conformance suite job fails 3 truncate-morphotactic fixtures (truncate-morphotactic-copy-u,-identity,-subsumption). Both name the truncate-morphotactic family that #480 already treats as nondeterministic, not these grammars; not compared against #480's own CI.Adds 18 HermitCrab stress grammars in a new
conformance-stress/directory, a sibling ofconformance/, for optimization work. They are not conformance fixtures and do not run by default. Each models a language type where one analysis or synthesis optimization matters: Bantu slot templates, agglutinative early-lexicon roots, dense co-occurrence, MPR concord, environment-conditioned allomorphs, and others. Expected parses are master's output, so a fixture is a parity check: an optimization must leave parses unchanged. Work is compared in step counts, not time.The unknown a reviewer arrives with is "does this change CI or the conformance numbers?". It should not: nothing that enumerates fixtures scans
conformance-stress/, and no workflow selects it.Where to look
tests/SIL.Machine.Morphology.HermitCrab.Tests/StressFixtureParityTests.cs:[Explicit]and[Category("Stress")], so a plaindotnet testreports it not executed. Opt in withdotnet test tests/SIL.Machine.Morphology.HermitCrab.Tests --filter TestCategory=Stress.--fixtures conformance-stress --include-pathological; the conformance runner needs--fixtures <path>and is otherwise pointed atconformance/.GrammarCoverageGate, ledgers, manifests and counterfactual coverage readconformance/only; no workflow (ci.yml, conformance.yml, counterfactual-coverage.yml) mentionsconformance-stress.words.yamlfiles starts with# kind: stress.Not here:
polundanasakinearly-lexicon-agglutinative(engines before #548, merged, trace an extra phonological rule for it); a trace-memory fixture with up to 864 parses per word; any marking of the flaky truncate fixtures (#480 has expect_fail, expect_skip and expect_crash, and no known-nondeterministic state).Verification: parity runs and the plain
dotnet testand runner comparisons against #480's head were done on the earlier 16-grammar version; they were not re-run for the 18-grammar head, so no pass counts are claimed here. csharpier and comment hygiene were clean then, and the comment-hygiene jobs pass now.Next: confirm
conformance-stress/as an opt-in directory here, and settle #480 so this can leave draft.Reading this a year from now -- start here
These grammars exist so an analysis or synthesis optimization can be checked for unchanged parses and measured in steps, without adding to the conformance suite's runtime or its coverage ledgers. The first 16 were added together;
bantu-tam-fusion(built for the impure-slot look-ahead stacked on #576) andtruncating-slot-chain(the worst case for the supply-set look-ahead stacked on #579) were added after. Both new grammars are synthetic, not real languages.Decisions and why
conformance/: the coverage gate, ledgers, manifests and counterfactual coverage all enumerateconformance/, so nothing there has to learn to skip these.[Explicit]plus a category: the test reusesFixture.DiscoverAllandRunner.RunSelfCheckfrom the conformance runner, so no new harness.Paths not taken
Marking the flaky truncate fixtures was left out because #480 has no known-nondeterministic state. A trace-memory fixture was left out because its words have up to 864 parses each.
Deferred
Whether the red Build and Conformance jobs match #480's own was not checked against #480's CI. A known-nondeterministic state in the conformance framework would let the truncate family stop failing CI; that belongs to #480.
🤖 Generated with Claude Code
This change is